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PREFACE 


This  report  documents  research  conducted  under  Project  AIR  FORCE  (former- 
ly Project  RAND)  by  The  Rand  Corporation.  The  work  described  here  was  per- 
formed as  part  of  the  project  entitled  "Analysis  of  Systems  for  Air  Force  Education 
and  Training”  under  Rand’s  Manpower,  Personnel,  and  Training  Program.  It  is  the 
first  in  a series  presenting  Rand’s  MODIA  planning  system.  MODIA,  a Method  Of 
Designing  Instructional  Alternatives,  is  a system  of  people,  computer  programs, 
and  procedures  that  allows  the  rapid  specification  and  simulation  of  courses  of 
instruction  during  the  early  stages  of  instructional  design.  It  augments  and  can  be 
used  in  the  present  Air  Force  instructional  systems  development  process. 

The  development  of  MODIA  has  been  supported  by  the  Deputy  Chief  of  Staff/ 
Personnel,  Headquarters  United  States  Air  Force,  and  the  Air  Training  Command, 
especially  DCS/Technical  Training,  the  Training  Development  Directorate,  and 
personnel  at  the  Keesler  School  of  Applied  Aerospace  Sciences.  It  is  part  of  Rand’s 
continuing  research  effort  in  the  areas  of  planning  and  management  in  education, 
education  technology,  and  the  cost  and  effectiveness  of  education  systems. 

The  series  of  MODIA  reports  includes: 

R-1700-AF,  MODIA:  Vol.  1,  Overview  of  a Tool  for  Planning  the  Use  of  Air 

Force  Training  Resources,  Polly  Carpenter-Huffman. 

R-1701-AF,  MODIA:  Vol.  2,  Options  for  Course  Design,  Polly  Carpenter- 

Huffman. 

R-1702-AF,  MODIA:  Vol.  3,  Operation  and  Design  of  the  User  Interface, 

Polly  Carpenter-Huffman  and  Ray  Pyles. 

R-1703-AF,  MODIA:  Vol.  4,  The  Resource  Utilization  Model  for  Instruc- 
tional Course  Design,  Margaret  Gallegos. 

R-1704-AF,  MODIA:  Vol.  5,  A User’s  Guide  to  the  Cost  Model,  Ron  Hess  and 

Phyllis  Kantar. 

This  report  introduces  the  series  by  telling  the  reader  in  general  terms  why 
MODIA  was  developed,  what  it  does,  and  what  it  is. 

MODIA  is  the  product  of  many  people.  The  author  is  especially  indebted  to 
Bernard  Rostker,  director  of  Rand’s  Manpower,  Personnel,  and  Training  Program, 
whose  unflagging  support  has  made  MODIA  possible.  Ray  Pyles  contributed  the 
discussion  of  MODIA’s  relation  to  steps  in  planning,  and  suggestions  from  Robert 
Rippey,  Philip  Doughty,  and  Helen  Turin  improved  the  clarity  and  completeness 
of  this  report.  Marilyn  La  Prell  typed  the  original  manuscript.  Sally  Belford  pre- 
pared the  pasteup  for  printing. 
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I.  INTRODUCTION 


PURPOSE  AND  ORGANIZATION  OF  THIS  REPORT 

MODIA  is  a complex  system;  the  subsequent  reports  describing  it  are  detailed 
and,  at  times,  technically  demanding.  This  report  tells  the  reader  in  general  terms 
why  MODIA  was  developed,  what  it  does,  and  what  it  is.  After  this  Introduction, 
which  discusses  the  why  of  MODIA,  Sec.  II  describes  MODIA  from  several  points 
of  view.  Section  III  presents  an  example  of  its  application  to  a training  course.  The 
last  section  discusses  the  evaluation  of  MODIA  by  the  Keesler  School  of  Applied 
Aerospace  Sciences. 


THE  PROBLEM 

The  Air  Force  is  heavily  involved  in  training;  in  peacetime  almost  all  Air  Force 
activities  may  be  considered  as  training  of  one  kind  or  another.  The  most  visible 
and  highly  structured  of  these  acti  'ities  is  conducted  by  Air  Training  Command 
(ATC)— Basic  Military  Training,  Fly  ng  Training,  and  Technical  Training— a multi- 
billion dollar  enterprise  requiring ( re  support  and  involvement  of  over  12  percent 
of  Air  Force  personnel. 

The  largest  single  component  of  ATC  is  devoted  to  formal  Technical  Training, 
which  prepares  Air  Force  personnel  for  jobs  ranging  from  aircraft  maintenance  to 
personnel  administration.  In  1976,  some  150,000  persons — over  a quarter  of  the 
force — were  scheduled  to  graduate  from  formal  courses  given  in  established  techni- 
cal schools. 1 The  operating  cost  of  this  activity  was  to  be  over  $600  million;  9 percent 
of  Air  Force  personnel  were  to  be  engaged  in  such  training  at  any  particular  time. 
Because  of  its  large  student  load,  formal  Technical  Training  offers  opportunities  for 
realizing  large  dollar  savings,  even  though  such  savings  are  small  in  terms  of  the 
individual  student. 

Many  opportunities  to  improve  the  management  of  training  resources  arise  in 
the  normal  course  of  events.  There  are  currently  some  300,000  different  course 
hours  in  the  curriculum,  of  which  over  a third  are  substantially  revised  or  newly 
prepared  annually.  Changes  in  force  composition,  introduction  of  new  weapon 
systems,  and  changes  in  operating  policies  of  other  commands  all  affect  the  subject 
matter  of  training  and  can  have  direct  effects  on  requirements  for  training  equip- 
ment and  indirect  effects  on  training  operations.  Shifts  in  training-related  char- 
acteristics of  the  student  population  (such  as  general  academic  ability  or  previous 
experience  related  to  the  subject  matter  of  the  course)  may  require  changes  in 
teaching  method  or  shifts  in  subject  matter  emphasis.  Changes  in  school  policy 
toward  classroom  management  may  encourage  the  replacement  of  familiar  meth- 
ods with  new  teaching  materials  or  techniques.  Finally,  variations  in  requirements 
for  the  output  of  graduates  obviously  and  strongly  affect  the  availability  of  and 
requirements  for  training  resources. 

' Derived  from  Military  Manpower  Training  Report  for  FY  1977,  Department  of  Defense.  March 
1976,  pp  MO  and  B-9 
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Clearly  the  design  and  redesign  of  courses,  if  accomplished  more  effectively, 
could  lead  to  substantial  improvements  in  training.  The  current  Air  Force  approach 
to  course  design,  termed  Instructional  System  Development  (ISD),  is  outlined  in 
AFM  50-2. 2 ISD  is  a systematic  procedure  for  relating  the  content  and  conduct  of 
training  to  needs  in  the  field.  This  procedure  consists  of  five  steps: 

1.  Analyze  system  requirements — determine  what  tasks  should  be  per- 
formed on  the  job. 

2.  Define  education  or  training  requirements — determine  how  and  where 
performance  of  these  tasks  will  be  learned. 

3.  Develop  the  objectives  and  tests  for  instruction. 

4.  Plan,  develop,  and  internally  validate  instruction. 

5.  Conduct  instruction  and  evaluate  its  effectiveness  both  internally  and  in 
the  field. 

In  carrying  out  these  steps,  training  developers  are  guided  by  the  general  principles 
stated  in  AFM  50-2,  tempered  by  their  own  judgment  and  past  experience  and  by 
existing  school  policies  and  procedures.3 

Such  expertise  is  requisite  to  skillful  application  of  ISD,  but  course  planners 
have  lacked  assistance  in  two  key  areas.  First,  they  have  had  no  way  to  examine 
the  requirements  for  training  resources  implicit  in  a particular  course  design. 
Instead,  to  estimate  resource  requirements,  they  have  had  to  use  planning  factors 
(e.g.,  the  average  student-to-instructor  ratio)  based  on  past  school  experience.  As 
a result,  resource  requirements  have  entered  the  design  process  only  in  a gross, 
subjective  fashion  or  after  course  design  was  completed.  The  demands  for  book- 
keeping and  computation  attendant  on  constructing  and  costing  a course  design  has 
meant  that  only  rarely  has  more  than  one  design  been  considered  during  planning. 

Second,  they  have  not  been  able  to  explore  the  effects  of  variations  in  teaching 
strategy  on  course  design  in  a systematic  and  comprehensive  fashion.  Rather, 
changes  in  teaching  strategy  have  been  constructed  bit  by  bit  during  the  production 
of  the  course.  The  effects  of  these  changes,  unfolding  as  the  revised  course  has  been 
put  into  operation,  have  often  been  unforeseen.  The  inability  to  uncover  such 
possibilities  during  the  planning  stage  ..as  discouraged  the  design  of  courses  that 
depart  significantly  from  past  practice. 


PURPOSE  OF  MODIA 

MODIA  was  developed  to  help  the  Air  Force  manage  resources  for  formal 
training  by  systematically  and  explicitly  relating  quantitative  requirements  for 
training  resources  to  the  details  of  course  design  and  course  operation  during  the 
planning  stage.  Course  design  includes  selection  and  sequencing  of  subject  matter- 
placement  and  description  of  tests;  deciding  on  teaching  methods  and  the  roles  of 
instructors  and  other  training  personnel;  the  assignment  of  media,  training  equip- 
ment, and  facilities;  and  specifying  the  characteristics  of  the  trainees  and  policies 

* Instructional  System  Development,  Department  of  the  Air  Force,  AF  Manual  50-2.  .'il  July  1975. 

’Quantitative  approaches  to  training  development  are  partly  incorporated  at  step  1.  in  establishing 
the  tasks  commonly  performed  in  the  field,  and  at  steps  4 and  5.  in  internal  validation  and  field 
evaluation. 
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for  the  management  of  student  progress.  Course  operation  simulates  the  way  stu- 
dent progress  through  the  course  and  the  resulting  requirements  for  and  use  of 
training  resources  would  be  affected  by  these  design  decisions.  Thus  MODI  A (1) 
suggests  approaches  to  course  design  not  incorporated  in  available  planning  factors 
and  (2)  describes  the  effects  of  course  design  on  resource  use.  This  encourages 
course  developers  to  consider  alternative  designs;  as  the  acronym  implies,  the 
consideration  of  alternatives  is  MODIA’s  primary  objective. 

MODIA  is  not  a prescription  for  training,  nor  is  it  an  optimizing  model;  rather, 
it  is  neutral  with  regard  to  the  training  effectiveness  of  a course  design  in  terms 
of  student  learning  or  with  regard  to  the  desirability  of  a course  design  in  terms 
of  training  policy.  Although  it  does  not  prescribe  a particular  course  design  and  is 
neutral  with  regard  to  training  effectiveness,  MODIA  insures  that  planners  make 
all  the  design  decisions  necessary  for  rational  planning.  Although  it  does  not  opti- 
mize, MODIA  gives  planners  the  data  they  need  to  make  their  own  judgments  in 
comparing  alternative  plans.  Through  an  interactive,  iterative  process  it  encour- 
ages planners  to  consider,  for  example: 

• The  implications  of  the  subject  matter  for  requirements  for  training  re- 
sources and  teaching  strategy; 

• Characteristics  of  students  that  affect  learning  and  instruction; 

• The  effects  of  course  management  policies  and  teaching  strategy  on  learn- 
ing and  on  the  use  of  training  resources;  and 

• How  changes  in  one  element  of  course  design  will  affect  the  others. 

MODIA  has  been  designed  primarily  for  the  use  of  the  five  ATC  technical 
schools,  which  account  for  over  90  percent  of  the  student  load  in  technical  courses. 
Each  school  has  several  departments,  and  each  department  deals  with  a major 
subject  area  and  has  several  branches  that  are  responsible  for  the  training  in  a 
related  group  of  courses.  MODIA  is  directed  to  the  course  level  because  a student 
usually  takes  only  one  course  at  one  school  to  qualify  for  his  initial  |ob  assignment. 

Although  MODIA  has  been  designed  for  use  by  course  planners,  its  usefulness 
is  not  restricted  to  the  school  level.  On  the  contrary,  MODIA  would  be  very  helpful 
to  higher  headquarters  for  examining  the  implications  for  the  specifics  of  school 
operation  of  broad  changes  in  training  policy. 

The  most  fruitful  applications  of  MODIA  will  probably  be  in  Step  4 of  ISD — in 
the  planning  and  development  of  instruction.  However,  like  ISD  itself.  MODIA  can 
be  applied  at  any  of  several  stages  of  planning.  For  example,  MODIA  does  not 
require  that  all  objectives  and  tests  be  stated  in  criterion-referenced  terms  or  even 
that  all  be  identified  before  it  can  give  insight  into  course  development.  As  with  the 
steps  in  ISD,  among  which  feedback  and  interaction  should  refine  and  improve  the 
ultimate  result,4  MODIA  should  be  applied  at  different  levels  of  generality.  For 
example,  in  early  stages  of  planning  MODIA  may  show  that  with  a given  student 
load  there  is  not  enough  training  equipment  for  each  student  to  have  sufficient 
practice  on  it.  This  might  suggest  that  some  of  the  equipment-oriented  objectives 
could  be  redirected  toward  less  expensive  mockups,  computer  simulation,  or  other 
acceptable  substitutes.  Thus,  MODIA  has  numerous  slots  for  descriptive  data  that 
do  not  all  have  to  be  filled  accurately  before  results  can  be  useful.  Moreover, 

* A KM  50-2,  p 1-2. 
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II.  DESCRIPTION  OF  MODIA 


This  section  describes  MODIA  from  several  points  of  view.  Of  most  immediate 
interest  to  potential  users  of  the  system  are  its  requirements  for  inputs  and  its 
outputs,  answers  to  the  questions:  "What  do  I have  to  know  to  use  the  system?” 
and  "What  do  I get  out  of  it?”  Next,  the  components  of  the  system  and  the  relation- 
ships among  them  are  described.  The  role  of  each  component  in  planning  is  then 
discussed. 


INPUTS  AND  OUTPUTS 

MODIA’s  major  requirements  for  input  are  specification  of: 

. Whether  the  course  will  require  unusually  expensive  or  scarce  resources. 

• What  course  content  is  and  how  it  should  be  sequenced,  especially  with 
regard  to  those  portions  requiring  the  use  of  unusual  resources. 

. The  maximum  and  minimum  numbers  of  students  that  can  be  expected  to 
take  the  course  during  the  period  of  interest. 

• Whether  tests  will  affect  student  progress  in  the  course  and  at  what  points 
they  will  be  given. 

• How  each  unit  of  the  course  will  be  taught. 

• How  long  it  will  take  to  teach  the  course,  especially  those  parts  of  it  that 
require  unusual  resources.  This  implies  that  the  user  is  familiar  with  the 
content  of  the  course  and  characteristics  of  the  students  who  will  be  taking 
it,  and  has  some  idea  of  how  long  it  takes  students  with  those  character- 
istics to  learn  the  given  type  of  content  when  instructed  by  the  methods 
the  user  chooses  to  specify.  This  is  MODIA’s  most  demanding  requirement 
for  input. 

• The  percentage  of  entering  students  who  may  be  eliminated  from  the 
course  or  required  to  repeat  portions  of  it. 

• The  availability  and  unit  cost  of  resources  used  by  the  course. 

The  more  accurately  the  foregoing  are  known,  the  more  useful  MODIA's  out- 
put will  be.  Even  where  knowledge  is  lacking,  MODIA  can  be  used  to  test  the 
sensitivity  of  course  operation  and  cost  to  variations  in  the  unknown  quantities. 

From  the  inputs  described  above,  MODIA  derives  the  following  outputs: 

• Average  and  peak  student  loads. 

• Average  time  before  graduation  or  elimination  from  the  course. 

• Time  during  which  students  are  waiting  for  other  students  (when  students 
are  taught  in  groups  larger  than  one)  or  for  resources  (when  resources  are 
limited). 

• The  percent  of  time  each  resource  is  used. 

• The  number  of  resources  required,  if  they  are  furnished  as  needed. 
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• The  capacity  required  of  a resource,  if  it  is  unknown  and  to  be  furnished 
as  needed. 

• The  start-up,  annual,  and  five-year  cost  of  the  course. 


COMPONENTS  OF  MODIA 

People  are  the  most  important  component  of  MODIA.  As  with  any  tool. 
MODIA’s  product  is  only  as  good  as  the  user  can  make  it.  Skillful  application  of 
MODIA  requires  not  only  the  expertise  described  in  the  requirements  for  input  but 
thorough  familiarity  with  the  system  and  imagination  in  its  application.  Since  it  is 
highly  unlikely  that  all  of  these  qualities  will  be  found  in  a single  person,  "the  user" 
will  usually  be  several  people,  each  bringing  his  own  special  skills  and  experience 
to  bear  on  the  problem  of  course  design. 

Because  the  bulk  of  MODIA  resides  in  computer  programs,  users  may  in  a very 
short  time  generate  a blueprint  for  instruction  and  estimates  of  the  resources 
needed  to  produce  and  operate  the  resulting  training  course.  More  important, 
computerization  encourages  users  to  design  and  compare  alternative  plans  before 
the  course  specified  by  a particular  plan  is  produced  and  put  into  operation,  often 
a long  and  expensive  process.  Once  a baseline  course  design  has  been  constructed, 
alternatives  can  be  generated  in  a matter  of  hours  or  minutes,  depending  on  the 
degree  to  which  they  depart  from  the  baseline  and  the  richness  of  the  baseline 
design. 

MODIA  has  four  components,  the  description  of  options  for  course  design,  the 
User  Interface  ( UI ),  the  Resource  Utilization  Model  (RUM),  and  the  Cost  Model 
(MODCOM).  Figure  1 shows  the  interactions  between  the  user  and  these  compo- 
nents. Note  that  MODIA  has  two  main  points  for  entering  data — the  UI  and  MOD- 
COM— rather  than  automatically  translating  RUM  output  into  course  cost.  This  is 
because  decisions  concerning  costing  procedures  and  policies  are  often  contingent 
on  course  operation.  The  additional  entry  point  also  permits  planners  to  refine  the 
design  for  preferred  course  operation  before  undertaking  a complete  cost  analysis. 

The  separation  of  entry  points  has  a further  advantage;  most  of  the  information 
that  is  unique  to  ATC  is  captured  in  MODCOM,  so  that  the  UI  and  RUM  are  useful 
for  planning  training  in  a wider  range  of  institutional  settings.  Thus,  the  UI  and 
RUM  can  give  insights  for  planning  parts  of  courses,  rather  than  full  courses,  or 
courses  given  in  the  Air  Force  Academy,  the  Air  University,  or  other  Air  Force 
agencies.  Conversely,  ATC  can  use  MODCOM  independently  of  the  UI  and  RUM 
to  analyze  course  cost  without  requiring  access  to  the  considerably  larger  resources 
(computer  capacity  and  MODIA  analysts)  required  to  support  the  UI  and  RUM. 

The  Options  for  Course  Design  (Vol.  2)  provides  an  overview  of  the  data  and 
information  the  UI  will  ask  for,  the  range  of  choices  available  at  each  entry  point, 
and  the  pros  and  cons  of  each  choice  as  they  affect  course  operation,  cost,  or 
instructional  effectiveness. 

The  UI  is  an  interactive  computer  program;  that  is,  the  user  enters  data  step 
by  step  in  response  to  questions  from  the  computer.  The  choice  of  question  the 
computer  asks  at  a given  point  is  influenced  by  preceding  responses  from  the  user, 
hence  the  term  "interactive."  Also,  at  many  intermediate  points,  the  computer 
processes  the  set  of  answers  given  to  that  point  and  displays  the  results  to  guide 


Fig.  1 —Interactions  between  the  user 
and  MODIA  components 
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further  decisionmaking  or  to  allow  the  user  to  recycle  through  the  process  if  he  is 
dissatisfied  with  the  results  at  that  point.  In  this  way  the  UI  produces  a course 
description  in  computer-compatible  data  that  interrelates  course  content,  teaching 
strategy,  student  characteristics,  and  resource  assignments. 

MODIA  inputs  these  data  automatically  to  the  RUM,  which  simulates  the  way 
in  which  student  progress  through  the  course  generates  requirements  for  training 
resources.  The  RUM  is  a "batch  process”  program:  It  receives  all  of  the  inputs  in 
a single  batch,  not  step  by  step  as  in  an  interactive  program.  It  also  produces  its 
outputs  in  a single  batch.  The  outputs  are  detailed  reports  on  course  operation 
including  student  flow  patterns  and  waiting  times  as  well  as  resource  demand  and 
use. 

Planners  will  rarely  be  satisfied  with  the  results  of  the  first  complete  operation 
of  the  UI  and  RUM  and  will  repeat  the  process  several  times  before  they  prepare 
the  input  required  for  MODCOM.  They  may,  however,  use  MODCOM  at  an  early 
stage  to  compare  rough,  order-of-magnitude  cost  estimates  to  help  them  select  from 
among  preliminary  course  designs. 

MODCOM  estimates  the  five-year  investment  and  operating  cost  associated 
with  a given  course  design.  It  is  also  a batch  process  program,  but  its  inputs  are 
provided  by  the  planner.  He  can  take  some  inputs,  such  as  resource  requirements, 
from  RUM  output,  but  he  must  draw  on  other  sources  for  such  items  as  equipment 
cost.  He  may  also  want  to  override  cost  and  manning  factors  stored  in  the  program. 
MODCOM  provides  options  as  to  which  costs  should  be  included  and  how  they 
should  be  computed,  and  supplies  outputs  in  two  formats,  one  relating  costs  to 
training  resources  (instructors,  students,  equipment,  etc.)  and  one  to  categories  in 
the  Air  Force  program  budget. 

Since  MODCOM  requires  some  input  that  is  not  a direct  consequence  of  RUM 
output,  planners  may  wish  to  exercise  MODCOM  several  times  without  revising 
initial  input  to  the  UI.  Or  the  course  cost  reports  may  highlight  a feature  of  course 
operation  that  is  unwarrantedly  expensive,  indicating  additional  operation  of  the 
UI  and  RUM.  Since  subsequent  passes  rarely  entail  complete  redesign  of  the 
course,  they  often  take  only  a small  fraction  of  the  time  and  attention  required  for 
creation  of  the  first  case. 

When  the  planners  are  satisfied  with  both  the  course  operation  and  course  cost 
reports  for  a specific  course  design,  they  have  at  hand  the  bulk  of  the  elements  for 
a training  plan  and  need  only  synchronize  the  plan  with  other  planning  activities 
at  the  school  to  put  it  in  final  form. 


RELATION  OF  COMPONENTS  TO  STEPS  IN  PLANNING 

MODIA  has  been  designed  to  be  an  assistant  to  planners,  encouraging  them  to 
consider  many  alternative  designs  in  the  process  of  planning  the  course.  MODIA 
performs  time-consuming,  tedious,  and  complex  tasks  that  are  performed  well  by 
a computer— to  calculate,  to  verify,  to  categorize,  and  to  simulate.  People  are 
required  to  perform  tasks  that  are  done  poorly  by  the  computer — to  conceive,  to 
design,  and  to  evaluate.  MODIA  assists  in  these  functions  by  structuring  the  design 
process,  suggesting  alternative  choices,  and  estimating  the  effects  of  choices  on 
course  operation  and  cost. 
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Course  planning  may  be  thought  of  as  three  operations.  Given  the  objectives 
for  a particular  course,  first  a design  for  that  course  may  be  synthesized,  then  *hat 
design  may  be  analyzed  to  determine  its  cost  and  operating  characteristics,  and 
finally  the  design  may  be  evaluated  against  some  desired  goal  or  other  alternative 
designs.  Figure  2 shows  the  relation  of  MODIA’s  components  to  these  steps.  The 
Options  for  Course  Design  and  the  UI  aid  in  the  synthesis  of  the  course  design,  and 
the  RUM  and  MODCOM  constitute  the  major  portion  of  the  second  function,  to 
analyze  the  course  design.  The  planner  performs  the  evaluation. 

In  the  synthesis  of  alternative  designs,  planners  may  attempt  to  attain  certain 
educational  objectives  within  the  constraints  of  school  policy,  the  resources  on 
hand,  budget  available  to  buy  new  resources,  types  and  kinds  of  instructors  avail- 
able, and  types  and  numbers  of  students  expected.  They  must  combine  these  factors 
with  knowledge  of  the  subject  matter,  some  educational  psychology,  and  some 
common  sense  to  arrive  at  a design  for  the  course.  Within  these  constraints,  they 
have  considerable  freedom  to  design  the  course  they  believe  will  best  attain  the 
educational  objectives.  For  example,  they  can  determine  the  type  of  material  to  be 
presented,  its  sequence,  the  mechanisms  for  presenting  it,  and  which  students  will 
receive  what  material.  Such  a design  process  is  largely  inductive,  subjective,  and 
innovative,  qualities  seldom  associated  with  computers. 

During  the  synthesis  of  alternative  designs,  each  decision  must  be  checked  to 
verify  that  there  are  no  internal  conflicts  among  decisions  made  at  different  points 
in  the  design  process  and  that  no  part  of  the  design  is  left  out.  For  example,  there 
must  be  an  instructional  policy  for  all  objectives  in  the  course  and  the  resource 
requirements  implicit  in  each  policy  must  have  been  considered.  Therefore,  the 
primary  roles  assigned  to  the  UI  for  synthesis  are  to  assure  that  each  alternative 
design  is  internally  consistent  and  complete,  and  to  translate  each  into  data  that 
are  compatible  with  the  programs  that  subsequently  analyze  the  course  design. 

The  analysis  of  alternative  designs  requires  careful,  detailed  estimation  of  the 
performance  of  the  course  in  several  respects,  including  training  effectiveness, 
graduation  rates,  failure  rates,  bottlenecks  in  student  progress,  and  cost.  In  most 
cases,  planners  currently  perform  the  analysis  manually  using  institutional  rules 
of  thumb,  personal  educational  wisdom,  and  a combination  of  intuition  and  inspira- 
tion. Although  MODIA  cannot  predict  the  instructional  effectiveness  of  the  course 
— the  ability  of  the  course  to  produce  students  with  a particular  set  of  skills — the 
RUM  can  estimate  its  operational  performance  and  MODCOM,  its  cost.  Thus,  the 
planner’s  knowledge  of  the  instructional  value  of  particular  training  methods  is 
augmented  by  MODIA’s  analysis  of  cost  and  operational  performance. 

Evaluation  of  the  alternative  designs  is  another  subjective  process  requiring 
the  weighing  of  the  various  performance  measures  to  select  an  acceptable  or  best 
course  design  from  the  alternatives.  There  is  no  single  best  way  to  do  this,  because 
different  schools  have  different  criteria  of  excellence.  For  example,  one  school  may 
be  particularly  concerned  with  the  cost  of  instructors  while  another  may  trade 
increased  cost  for  higher  performance.  Since  such  a rich  variety  of  considerations 
must  be  balanced  in  the  evaluation  of  alternative  designs,  it  seems  unlikely  that 
a computer  program  can  be  written  that  would  ultimately  select  the  "best"  course 
design  for  a particular  course.  Therefore,  this  entire  function  is  best  performed  by 
human  planners. 

One  tool  that  :s  frequently  proposed  to  assist  in  the  choice  among  alternatives 
is  "cost-effectiveness"  analysis.  In  general,  the  alternatives  designed  with  MODIA 
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will  vary  in  both  cost  and  effectiveness.  They  can  be  compared  with  absolute 
standards,  such  as  budgetary  constraints  or  minimum  required  instructional  effec- 
tiveness, and  with  each  other.  One  might  be  rejected  because  it  violates  the  stan- 
dards, another  because  it  is  similar  in  cost  (or  effectiveness)  to  other  alternatives 
but  of  much  less  effectiveness  (or  greater  cost).1  Where  there  is  no  clear-cut  ra- 
tionale for  choice,  the  alternative  will  be  chosen  that  has  the  greatest  subjective 
appeal  to  the  planner.  It  is  important  to  stress  that,  as  will  be  illustrated  in  the  next 
section,  comparisons  of  cost  or  effectiveness  should  not  be  made  among  designs  that 
have  different  levels  of  detail  regarding  comparable  elements,  such  as  the  specifica- 
tion of  resources.  Although  not  all  elements  need  be  defined  in  the  same  detail, 
comparisons  among  designs  with  different  levels  of  refinement  for  comparable 
elements  will  usually  not  be  valid.  In  other  words,  to  define  comparable  cases,  all 
elements  whose  cost  is  likely  to  differ  significantly  should  be  included. 


RESOURCES  REQUIRED  TO  SUPPORT  MODIA 

MODIA  is  currently  operating  in  two  installations,  the  Keesler  School  of  Ap- 
plied Aerospace  Science  near  Biloxi,  Mississippi,  and  The  Rand  Corporation  in 
Santa  Monica,  California.  Each  of  these  provides  a different  instance  of  how 
MODIA  can  be  supported.  For  example,  the  primary  MODIA  computer  at  Keesler 
is  the  Honeywell  6060,  whereas  at  Rand  it  is  the  IBM  370/158. 

Below  is  a discussion  of  the  resources  needed  to  support  MODIA  and  the 
minimum  standards  they  should  meet.  In  general,  these  standards  are  such  that 
components  that  must  be  readily  accessible  to  school  planners  can  be  supported  at 
any  of  ATC’s  technical  schools.  The  RUM,  which,  because  it  uses  SIMSCRIPT  II. 5, 
may  need  to  be  operated  remotely,  does  not  have  to  be  at  the  planner’s  fingertips. 

Personnel 

Competent  people  are  crucial  to  the  effective  use  of  MODIA.  Two  groups  should 
be  involved  in  any  application  of  MODIA  in  planning — those  who  are  expert  in 
using  MODIA  (hereafter  referred  to  as  "the  MODIA  team”)  and  those  who  have 
knowledge  and  experience  in  the  areas  of  subject  matter  and  planning  particular 
to  the  course  being  developed  (hereafter  referred  to  as  the  "subject  matter  ex- 
perts”). Figure  3 shows  the  configuration  currently  in  use  at  Keesler. 

Members  of  the  MODIA  team  need  not  (and  probably  should  not)  have  exten- 
sive experience  with  computers.  By  background  and  bent  they  should  be  problem 
solvers  first,  computer  experts  second.  They  need  to  be  familiar  with  course  plan- 
ning and  school  operations,  so  that  they  can  draw  out  the  subject  matter  experts’ 
best  judgment  on  what  constitutes  effective  instruction  and  help  them  distinguish 
between  what  is  usually  done  (for  convenience  or  by  tradition)  from  what  is  needed. 
They  should  also  have  a feeling  for  how  far  school  policies  can  be  adjusted  and  if 
it  would  be  desirable  to  do  so,  and  they  should  be  able  to  act  as  liaison  among 
different  organizational  entities  within  the  school  whose  interests  may  clash  within 
a given  training  course  design. 

1 For  a comprehensive  discussion  of  this  process  of  analysis,  see  Carpenter  and  Hapyart,  Cost 
Effectiveness  Analysis  for  Educational  Planning.  The  Rand  Corporation.  P-4327,  March  1970 
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Fig.  3— Configuration  currently  in  use  in  Keesler 
School  of  Applied  Aerospace  Science 


Members  of  the  MODIA  team  need  initial  training  in  the  use  of  the  system  and 
need  to  apply  the  system  frequently  enough  to  maintain  their  expertise.  They 
should  be  fully  aware  of  MODIA’s  features  and  operation,  particularly  of  the 
alternatives  it  encompasses;  should  have  a good  feel  for  the  effects  of  different 
choices  of  input  on  MODIA  outputs;  and  should  be  able  to  guide  the  subject  matter 
experts  in  choosing  the  most  efficient  alternatives. 

The  subject  matter  experts  are  people  who  normally  plan  and  develop  courses. 
They  need  an  understanding  not  only  of  the  subject  of  the  course  but  of  the  student 
population  and  the  teaching  methods  that  work  best  for  them.  They  should  be 
familiar  with  school  policy  and  should  know  what  resources  are  likely  to  be  avail- 
able to  the  course  and  what  they  will  cost.  All  of  this  information  need  not  reside 
in  a single  person  (nor  does  it  in  the  current  development  process).  Perhaps  most 
important,  the  subject  matter  experts  need  to  be  flexible — able  to  interact  with  the 
MODIA  team  so  that  the  special  capabilities  of  both  groups  can  be  fully  applied. 

Configuration  of  Hardware  and  Facilities 

Two  features  of  the  system  configuration  are  of  overriding  importance  to 
course  planners — responsiveness  and  accessibility.  Planners  should  be  able  to  use 
the  system  at  almost  any  time  during  the  working  day  so  they  can  fit  MODIA  in 
with  their  other  normal  duties,  and  they  should  have  access  to  MODIA  in  locations 
that  are  convenient  to  their  usual  places  of  work  and  free  from  undue  distractions. 
These  characteristics  are  particularly  crucial  to  the  use  of  the  UI,  which  ideally 
should  be  operated  from  a portable  terminal. 

To  facilitate  the  smooth  flow  of  the  design  process,  the  computer  should  process 
and  respond  to  inputs  to  the  UI  within  five  seconds  of  the  completion  of  input  for 
at  least  90  percent  of  such  completions.  Response  times  for  the  batch  programs 
(RUM  and  MODCOM)  need  not  be  so  short.  (Response  time  for  a batch  program 
is  the  time  from  job  submission  to  the  return  of  printed  output  to  the  user,  including 
communication  time  if  a remote  facility  is  used.)  Response  times  for  batch  pro- 
grams should  not  exceed  24  hours  for  95  percent  of  the  jobs  submitted.  Otherwise, 


13 


the  interactive,  iterative  nature  of  the  design  process  will  be  destroyed.  Of  course, 
it  is  highly  desirable  for  all  response  times  to  be  as  short  as  possible;  they  will 
normally  vary  depending  on  the  extensiveness  of  the  course  design,  the  operating 
system  and  hardware,  and  the  number  and  size  of  competing  jobs  being  performed 
on  the  computer. 

The  MODIA  hardware  and  facilities  should  be  arranged  in  two  groups— a 
computer  facility  and  one  or  more  working  areas.  In  the  computer  facility  are  all 
of  the  computer  hardware  except  the  interactive  terminal — the  computer,  card 
reader,  high-speed  printer,  and  other  supporting  hardware — while  the  working 
areas  are  outfitted  like  a normal  office  (with  desks,  chairs,  chalkboards,  filing  cabi- 
nets, storage  shelves,  telephones,  and  other  office  equipment  and  supplies)  with  the 
addition  of  the  interactive  remote  terminal.  Such  arrangements  allow  the  planners 
to  work  privately,  away  from  the  activity  of  the  computer  facility  for  much  of  the 
time  and  in  closer  contact  with  their  own  documents,  preliminary  course  plans,  and 
other  materials  necessary  to  the  planning  activity.  MODCOM  uses  keypunched 
card  input,  so  arrangements  will  also  be  needed  for  occasional  keypunch  support. 

Computer  system  requirements  for  supporting  MODIA  are  given  in  the  appen- 
dix. 
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III.  AN  APPLICATION  OF  MODIA 


The  most  direct  way  to  acquaint  people  with  MODIA  is  through  a concrete 
example.  This  example  illustrates  some  of  the: 

• inputs  MODIA  uses  and  outputs  it  produces; 

• ways  in  which  MODIA  stimulates  the  design  of  alternatives; 

• levels  of  refinement  in  training  plans  that  can  be  considered. 

The  example  does  not  by  any  means  demonstrate  all  of  MODIA’S  features.  More 
exhaustive  examples  are  developed  in  the  other  reports  in  this  series. 


BASIS  OF  THE  EXAMPLE 

The  example  is  based  on  the  first  major  section  (or  "block”)1  of  course 
3ABR30431,  Flight  Facilities  Equipment  Repairmen,  as  it  was  taught  at  the  Kees- 
ler  School  of  Applied  Aerospace  Sciences  in  the  fall  of  1973  and  described  by  Keesler 
personnel  who  were  expert  in  this  subject  matter.  The  example  is  carried  through 
the  other  reports  in  the  series. 

This  block  of  the  course  oriented  the  student  to  general  characteristics  of  the 
job  for  which  he  was  being  trained  and  taught  him  fundamental  skills  and  proce- 
dures for  maintaining  the  AN/URN-5  (a  low  frequency  beacon).  Almost  all  students 
who  entered  the  course  had  completed  an  18-week  course  in  electronic  principles 
(3AQR30020).  As  of  October  1973  the  course  for  Flight  Facilities  Equipment  Repair- 
men lasted  for  17  weeks,  so  that  the  total  training  for  those  who  graduated  was  35 
weeks.  Students  taking  this  course  must  have  scored  at  or  above  the  80th  percentile 
in  the  Electronic  Aptitude  battery  of  the  Airman’s  Qualifying  Examination.  Thus 
they  tended  to  be  above  the  average  airman  with  regard  to  both  general  academic 
ability  and  knowledge  of  electronics. 


ORIGINAL  BLOCK  DESIGN 
Training  Content 

The  goal  of  the  block  was  to  prepare  airmen  to  maintain  the  AN/URN-5  low 
frequency  beacon,  a ground-based  device  used  in  air  navigation.  Maintenance  con- 
sisted of  various  procedures  for  servicing  the  beacon,  troubleshooting  the  beacon 
when  it  malfunctioned,  checking  it  during  aircraft  flight,  and  installing  it  in  a flight 
facility.  Only  knowledge  of  procedures  for  the  last  two  tasks  were  taught,  rather 
than  ability  to  perform  them. 

The  example  includes  the  following  content  and  sequence,  and  the  time  re- 
quired for  instruction: 

' "Block"  is  ATC  terminology  for  a portion  of  a course  that  teaches  one  or  two  major  topics. 
Frequently  a block  deals  with  an  item  of  major  equipment,  as  in  this  example. 
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Test  Equipment.  The  trainee  was  familiarized  with  the  equipment  used  to 
service  and  test  the  AN/URN-5  (2-1/2  hours). 

System  Familiarization.  The  trainee  learned  what  the  components  of  the 
AN/URN-5  were  and  how  to  read  the  schematics  associated  with  it  (8  hours). 

Servicing.  The  trainee  learned  to  perform  a simple  procedure  for  servicing  the 
beacon  (1/2  hour). 

Schematics.  The  trainee  learned  to  analyze  more  detailed  schematics  for  the 
system  (6  hours). 

Servicing.  The  performance  of  more  complex  procedures  for  servicing  the 
AN/URN-5  was  taught  next  (1-1/2  hours). 

Principles  of  Troubleshooting.  Fundamental  principles  to  be  applied  in  trou- 
bleshooting electronic  equipment  were  taught  (1-1/2  hours). 

Troubleshoot  AN/URN-5.  The  trainee  learned  to  find  the  cause  of  malfunc- 
tions induced  by  the  instructor  (1-1/2  hours). 

Flight  Checking.  The  trainee  learned  what  procedures  were  used  for  checking 
beacon  operation  during  aircraft  flight  (3/4  hour). 

Installation.  The  trainee  learned  what  procedures  were  used  for  installing  the 
beacon  in  a flight  facility  (3/4  hour). 

Written  Test.  A test  was  given  emphasizing  the  trainee’s  ability  to  read  sche- 
matics (1/2  hour). 

Practical  Test.  Next,  the  trainee  was  tested  on  his  ability  to  perform  one  of  the 
servicing  procedures  (1/2  hour). 

Critique.  The  block  ended  with  the  instructor  discussing  test  results  with  the 
class  (1/4  hour). 

Teaching  Method 

The  instructor  used  the  discussion-demonstration  method  to  teach  the  facts  and 
concepts  required;  this  classroom  work  was  supplemented  by  assignments  for  home 
study.  He  spent  the  largest  share  of  his  time  guiding  student  practice  in  reading 
schematics,  also  supplementing  the  activity  with  home  study.  Students  performed 
on  the  equipment  (servicing  and  troubleshooting)  in  teams  of  two,  with  the  instruc- 
tor providing  guidance  as  needed. 

Student  Entries  and  Repeats 

A maximum  of  eight  students  was  expected  to  enter  each  week.  From  past 
experience,  Keesler  subject  matter  experts  expected  15  percent  of  the  students  to 
do  so  badly  on  the  test  that  they  would  have  to  take  the  block  over  again.  No 
student  would  be  eliminated  from  the  course  on  the  basis  of  test  results. 

Resource  Characteristics 

Major  resources  used  were  the  instructor,  the  AN/URN-5,  a classroom  for 
general  instruction,  and  a lab  housing  the  equipment. 

Instructor.  The  instructor  could  handle  up  to  ten  students  in  conventional 
classroom  work  but  could  supervise  only  four  during  performance  on  equipment. 
Thus,  at  times  three  instructors  would  be  needed  during  student  performance  to 
accommodate  students  repeating  the  block. 
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AN/URN-5.  Only  two  students  could  work  on  one  low-frequency  beacon  at  a 
time.  Therefore,  five  beacons  would  be  needed,  again  to  accommodate  repeating 
students. 

Classroum.  This  would  be  a standard  classroom  with  a capacity  for  ten  stu- 
dents. 

Lab.  The  lab  would  contain  the  five  beacons  and  space  for  up  to  ten  students 
to  work  on  them. 


MODIA  OUTPUTS  FOR  ORIGINAL  BLOCK  DESIGN 

The  above  information  was  entered  in  the  UI  and  MODCOM.  With  regard  to 
course  operation,  MODIA  shows  that  although  the  block  length  is  just  over  24 
hours  (24  hours  and  15  minutes),  the  average  time  a student  spends  in  the  course 
is  more  like  32-3/4  hours  because  of  the  relatively  high  percentage  of  students  who 
must  repeat  the  block  and  because  repeating  students  have  to  wait  to  join  classes 
entering  later.  Similarly,  although  the  average  student  load  would  be  6.4  students 
(because  the  block  length  is  shorter  than  the  entry  interval)  without  student  re- 
peats, it  is  actually  nearly  nine  students  because  of  the  repeats.  Finally,  because 
resources  have  been  supplied  to  meet  peak  demands,  which  occur  during  only  a 
small  portion  of  the  course  (student  performance  on  equipment),  most  resources  are 
used  very  little.  For  example,  the  low-frequency  beacons  are  used  only  12  percent 
of  the  time,  and  the  instructors  are  occupied  only  34  percent  of  the  time. 

Assuming  that  the  block  is  taught  in  this  way  for  a period  of  five  years  and  that 
all  resources  must  be  acquired  at  the  outset,  this  block  of  instruction  would  cost  at 
least2  S839  thousand  (not  counting  the  cost  of  overhead).  Given  the  underutilization 
of  resources  noted  above,  it  is  natural  to  ask  whether  a redesign  of  the  block  might 
result  in  better  use  of  fewer  resources  and,  hence,  a lower  cost. 


AN  ALTERNATIVE  DESIGN 


Although  an  objective  of  redesign  might  be  to  reduce  cost,  another  objective 
might  be  to  improve  the  training  effectiveness  of  the  block  at  the  same  time.  For 
example,  a special  track  might  be  formed  for  slower  students  so  that  the  instructor 
could  concentrate  his  attention  on  them  in  smaller  classes.  Bright  students,  who 
compose  the  bulk  of  trainees  in  this  career  field,  could  learn  mainly  through  self- 
instruction.  Letting  the  brighter  students  progress  at  their  own  pace  would  also 
have  the  effect  of  staggering  the  demands  for  resources  so  that  peak  demands 
would  occur  at  different  times  for  the  two  tracks;  this  could  solve  some  of  the 
resource  problems  posed  above. 

Specifically,  assume  that  40  percent  of  the  students  belong  in  the  slow  track 
where  they  are  instructed  by  an  instructor.  The  brighter  students  work  largely  on 
their  own  with  "scrambled”3  books  in  a study  area  within  the  laboratory,  where 

2 "At  least”  because  many  resources  that  would  be  required  have  not  been  include  in  this  estimate. 

3 A "scrambled”  book  presents  the  subject  in  small  units,  each  followed  by  a question  testing  the 
student's  grasp  of  the  unit  The  sequence  of  units  a student  reads  is  determined  by  his  choice  of  response 
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another  instructor  can  answer  questions  as  necessary.  It  takes  the  bright  students 
about  3 I as  long  to  complete  the  self-study  assignments  as  the  slow  students  spend 
with  the  instructor  learning  the  same  materials.  All  students  work  on  the  equip- 
ment in  tt . "is  of  two.  as  before,  with  the  instructor  responsible  for  monitoring  the 
work  of  no  more  than  two  teams  at  a time. 

To  establish  an  upper  bound  on  the  resources  required  for  this  alternative,  we 
first  ask  MODIA  to  compute  the  number  of  resources  required  if  they  are  to  be 
furnished  as  needed  Some  outputs  from  this  computation  are  shown  in  Table  1, 
along  w ith  data  on  the  base  case. 

The  five-year  cost  for  the  base  case  shown  on  Table  1 has  been  raised  to  $842,000 
(from  $839,0001  to  include  the  cost  of  development  and  procurement  of  printed 
materials  for  comparison  with  the  materials  for  self-study  used  in  the  alternative, 
which  are  more  expensive  to  develop.  This  illustrates  a general  principle:  To  define 
comparable  cases,  all  areas  where  costs  are  likely  to  differ  should  be  included. 

The  upper  bound  case  (column  2 in  Table  1)  was  not  costed  because  it  seemed 
likely  that  a less  expensive  case  could  be  designed  by  reducing  the  stock  of  re- 
sources that  are  used  very  little.  This  turned  out  to  be  true,  as  the  third  column 
shows,  where  the  number  of  low-frequency  beacons  has  been  reduced  from  four  to 
three.  Although  the  alternative  uses  the  same  number  of  instructors  as  the  original 
design,  it  uses  them  for  somewhat  more  of  the  time  (53  percent  rather  than  34 
percent).  This  is  because  of  the  way  students  tend  to  distribute  themselves  in 
self-paced  study. 

The  case  shown  in  column  3 is  somewhat  less  expensive  than  the  base  case, 
primarily  because  of  the  reduced  length  of  the  block.  Would  it  be  more  effective  in 
teaching  the  subject?  If  so,  perhaps  the  recycle  rate  would  actually  drop  enough  to 
permit  the  use  of  one  less  instructor.  This  would  result  in  another  7 percent  saving 
ever  the  base  case.  Whether  a sufficiently  large  drop  in  recycles  would  result  from 
the  changed  approach  can  only  be  established  in  actual  application. 


Table  1 

Comparison  of  Alternative  Block  Designs 


Base  Case 

MODIA 

Revised 

(unlimited 

resources) 

MODIA 

Revised 

(limited 

resources) 

Average  student  load 

8.7 

7.2 

7.2 

Peak  student  load 

13.0 

12.0 

12.0 

Average  time  to  finish  block 

32:44 

26:54 

26:58 

Resources 

Low-frequency  beacon 

Number 

5 

4 

3 

Percent  of  time  used 

12 

12 

16 

Instructors 

Number 

3 

3 

3 

Percent  of  time  used 

34 

53 

53 

Five-year  cost  (thousand*) 

$842 

(not 

costed) 

$789 

IV.  EVALUATION  OF  MODIA 


During  the  spring  of  1976,  personnel  of  the  Keesler  SAAS  evaluated  MODIA 
in  an  operating  Air  Force  setting.  In  this  section,  the  objectives  of  the  evaluation, 
the  procedures  used  to  carry  it  out,  and  the  results1  are  discussed. 


OBJECTIVES  AND  PROCEDURES 

The  objectives  of  the  evaluation  were  "to  determine  the  validity  of  the  course 
operation  and  course  cost  data  (MODIA)  generates,  to  assess  its  effectiveness  as  a 
planning  tool,  to  determine  its  implementation  and  potential  operating  costs,  and 
to  determine  to  which  types  of  formal  technical  training  the  system  can  be  ap- 
plied.”2 The  major  criterion  of  planning  effectiveness  was  to  be  that  courses  pro- 
duced with  MODIA  would  be  sufficiently  less  expensive  than  existing  courses  to 
offset  the  additional  cost  (if  any)  of  using  MODIA. 

These  objectives  were  established  by  personnel  at  Hq  ATC  and  at  Keesler; 
Keesler  designed,  directed,  analyzed,  and  reported  on  the  evaluation.  Although 
Rand  provided  assistance  and  suggestions  as  requested  before  the  evaluation  be- 
gan, Rand  did  not  participate  in  the  evaluation  beyond  helping  Keesler  operate  the 
computer  programs. 

At  about  the  time  MODIA  was  being  installed  at  Keesler,  a group  of  people  at 
the  Lowry  SAAS  began  designing  a new  course  to  train  technical  instructors  to 
apply  less  traditional  teaching  methods,  such  as  computer-assisted  instruction.  At 
Lowry's  request,  Keesler  included  this  course  in  the  evaluation  Thus,  MODIA  was 
also  applied  to  an  entirely  new  course  for  the  evaluation. 

The  formal  evaluation  began  after  the  Keesler  MODIA  team  had  become  famil- 
iar with  the  system  and  Rand  had  made  a major  revision  to  the  UI  at  their  request. 
The  first  step  was  to  establish  MODIA’s  validity  by  using  it  to  simulate  courses  and 
comparing  the  resulting  estimates  of  course  operation  and  cost  with  historical  data 
describing  the  same  characteristics  for  existing  courses.  Five  courses  were  chosen 
that  had  recently  been  revised  manually  through  application  of  ISD  and  were 
considered  products  of  high-quality  planning.  The  comparison  was  made  for  the 
courses  as  they  had  been  before  revision,  since  complete  cost  and  operation  data 
were  not  available  for  the  revised  courses.  A team  of  three  to  five  subject  matter 
specialists  worked  with  the  MODIA  team  to  insure  the  accuracy  of  the  simulation 
and  the  cost  factors  entered  into  MODCOM  for  each  course.  During  this  activity, 
careful  records  of  resource  consumption  (primarily  personnel  and  computer  time) 
were  maintained. 

Next,  each  course  as  it  had  been  revised  manually  was  simulated  and  costed 
using  MODIA  to  establish  a baseline  case  for  comparison  with  alternatives  de- 
signed with  the  help  of  MODIA.  Of  course,  no  baseline  was  available  for  comparison 

' Kmluation  of  t he  MODIA  System.  ATC  OK  76-1  L’SAF  School  of  Applied  Aerospace  Sciences. 
Keesler  AFB.  Mississippi.  July  30.  1076 
2 Ibid.  p.  i i. 
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with  designs  that  the  planners  produced  for  the  Lowry  course.  Two  alternatives 
were  prepared  in  addition  to  each  of  the  baseline  courses.  Again,  the  subject  matter 
specialists  worked  with  the  MODIA  team,  and  careful  records  were  kept  of  re- 
source use. 

Concurrently,  the  Keesler  evaluators  gathered  information  describing  the  mag- 
nitude of  the  Keesler  manual  planning  activity  in  terms  both  of  the  percentage  of 
total  course  hours  that  are  new  or  revised  annually  and  the  amount  of  personnel 
time  that  had  been  spent  in  manual  planning  of  the  five  baseline  courses.  At  the 
end  of  the  evaluation,  structured  interviews  of  the  MODIA  team  and  the  subject 
matter  experts  who  had  participated  in  the  evaluation  were  conducted  "to  deter- 
mine their  perception  of  MODIA’s  value  as  a planning  tool  and  to  identify  factors 
affecting  its  effectiveness.”3 


RESULTS 

For  four  of  the  courses  simulated  by  MODIA,  course  cost  and  operation  param- 
eters were  within  2 to  8 percent  of  those  reported  for  the  courses  before  manual 
revision.  (The  Keesler  Comptroller  considers  MODIA  cost  estimates  to  be  sufficient- 
ly valid  if  they  are  within  10  percent  of  the  comptroller’s  cost  figure.)  One  course 
(3AQR30020-1,  Electronics  Principles),  however,  was  outside  the  specified  range. 
This  course,  taught  by  the  self-paced  method,  has  a large  number  of  modules  of 
instruction  and  a very  large  student  load.  The  simulation  kept  track  of  each  student 
as  he  moved  through  each  module  at  an  individually  assigned  rate.  The  resulting 
mass  of  data  was  so  large  that  the  planners  could  simulate  only  600  hours  of  course 
operation,  rather  than  the  1500  hours  annually  devoted  to  this  course. 

For  three  of  the  baseline  courses  resulting  from  the  manual  planning  process, 
the  alternatives  generated  with  the  assistance  of  MODIA  were  less  expensive  than 
the  baseline  courses  as  simulated  in  MODIA.  For  one  course,  neither  of  the  alterna- 
tives designed  with  the  assistance  of  MODIA  was  less  expensive  than  the  baseline 
course.  For  another,  the  Electronics  Principles  course  discussed  earlier,  planners 
designed  an  alternative  using  MODIA  that  was  8 percent  (or  $4,000,000)  less  expen- 
sive than  the  baseline.  However,  both  the  baseline  and  the  alternative  were  derived 
by  extrapolation  from  a 600-hour  simulation,  so  the  validity  of  this  saving  is  open 
to  considerable  question. 

Although  the  Lowry  planners  devised  a course  design  with  MODIA’s  assistance 
that  they  felt  would  be  optimum,  they  also  were  unable  to  simulate  their  preferred 
design  as  accurately  as  they  wished.  Subsequently,  Rand  personnel  helped  them 
produce  a more  accurate  simulation  of  the  course  they  had  in  mind  so  that  they 
could  answer  several  of  their  questions  about  the  course  design. 

Manual  planning  is  not  comparable  to  planning  using  MODIA  because  MODI  A- 
assisted  planning  produces  several  alternatives,  whereas  in  manual  planning  the 
"perceived  single  best  course  design  is  pursued  from  the  beginning.”  Also,  manual 
planning  almost  always  represents  less  radical  degrees  of  revision  than  the  MODIA 
approach.4  Therefore  comparison  of  the  cost  of  planning  using  MODIA  with  the 
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• Ibid.  p.  22. 
' Ibid.,  p 28. 
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cost  of  manual  planning  is  not  valid;  however,  knowledge  of  MODI  A's  applicability 
and  costs  is  essential  in  deciding  whether  MODIA  is  worth  its  costs. 

More  than  half  of  the  hours  in  courses  to  which  MODIA  can  be  applied  at 
Keesler  are  new  or  revised  annually  (42,500  hours);  this  represents  80  percent  of 
Keesler’s  annual  planning  effort.5  Because  of  this  wide  applicability,  the  evaluators 
concluded  that  when  the  savings  that  the  evaluation  indicated  are  possible  in  the 
courses  tested  are  "projected  to  the  population  of  courses  to  which  MODIA  can  be 
applied”  they  outweigh  MODIA’s  operational  cost.6 

Perhaps  most  important  over  the  long  run  were  the  improvements  MODIA 
made  in  the  planning  process.  Personnel  involved  in  the  test  believe  that  MODIA 
will  enable  users  to; 

• Accurately  plan  new  and  revised  courses.  ISD  team  members  believed 
that  the  courses  simulated  by  MODIA  had  a great  deal  of  fidelity  to  the 
actual  courses  so  that  they  could  have  confidence  in  the  products  gener- 
ated by  MODIA  during  course  planning. 

• Systematically  design  a course  by  providing  guidance  that  assists  plan- 
ners in  making  the  major  design  decisions. 

• Rapidly  plan  instructional  alternatives  once  planning  data  have  been  col- 
lected and  worksheets  prepared.  It  required,  on  the  average,  three  days 
of  intense  over-the-shoulder  guidance  by  the  MODIA  team  for  each  ISD 
team  to  collect  and  translate  course  planning  data  to  MODIA  parameters 
for  system  operation.  However,  this  was  all  accomplished  in  about  one- 
fourth  the  time  required  to  plan  one  instructional  alternative  manually. 

• Consider  many  options  in  planning  instructional  alternatives. 

• Rapidly  compare  the  consequences  and  costs  of  alternative  decisions  con- 
cerning an  instructional  system.  From  MODIA  printouts,  team  members 
were  able  to  identify  areas  of  a course  (especially  in  resource  utilization) 
that  could  be  improved.  Although  they  had  to  use  their  own  judgment  to 
devise  the  changes  that  might  improve  the  course.  MODIA  quickly  gave 
them  feedback  on  the  consequences  of  these  changes.  Therefore,  in  an 
operational  environment,  planners  could  repeatedly  modify  the  simulated 
course  until  an  optimum  mix  of  instructors,  materials,  equipment,  facili- 
ties, and  sequence  of  learning  events  was  found.7 

• However,  it  is  apparent  that  enthusiastic  support  for  MODIA  will  evolve 
only  if  a competent  and  innovative  MODIA  Team  explains  the  system, 
provides  guidance  and  insights,  and  interprets  the  computer  products." 

The  evaluators  did  not  recommend  adoption  of  MODIA.  not  only  because  they 
believed  that  the  data  resulting  from  the  test  were  too  limited  to  do  so, but,  more 
important,  because  "It  has  not  yet  been  determined  how  SAAS  policies  and  objec- 
tives will  be  integrated  into  MODIA  planning,  and  at  what  level  the  products  of 
MODIA  will  be  used  in  decision  making. . . . Further,  neither  operating  procedures 
nor  the  necessary  operational  configuration  to  support  MODIA  planning  have  been 

5 Ibid.,  pp.  26  and  33. 

* Ibid.,  p.  32. 

1 The  above  material  paraphrases  points  made  in  Evaluation  of  the  MODIA  System,  p.  22 

■ Ibid.,  p.  33. 
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determined.”9  The  evaluators  concluded  by  recommending  a follow-on  evaluation 
to  "establish  both  the  procedures  and  organizational  configuration  necessary  to 
support  MODIA  planning,  and  using  realistic  management  problems,  establish  the 
utility  of  the  system  to  SAAS  management.”10 


SUMMARY 

The  Keesler  evaluation  of  MODIA  is  a valuable  contribution  to  the  literature 
on  planning  in  addition  to  being  a useful  report  on  the  strengths  and  weaknesses 
of  MODIA.  The  evaluation  revealed  that: 

• MODIA  performed  as  it  was  designed  to  do;  in  particular,  it  helped  plan- 
ners design  more  efficient  courses. 

• There  are  several  technical  ways  in  which  MODIA  can  be  made  more 
useful. 

• Personnel  costs  for  using  MODIA  are  fairly  low  but  the  computer  costs  are 
high.  These  can  be  more  than  offset  by  reductions  in  course  cost  that 
MODIA  encourages. 

. MODIA's  potential  value  to  ATC  hinges  on  personnel-related  issues, 
namely  the  insight  and  inventiveness  of  the  MODIA  team,  management 
procedures  for  using  MODIA  products,  and  the  incentives  for  school  plan- 
ners to  effect  savings  in  all  types  of  training  resources. 

9 Ibid.,  p.  33. 

10  Ibid. 


APPENDIX 

COMPUTER  SYSTEM  REQUIREMENTS 


Since  it  may  be  necessary  to  install  different  MODIA  programs  on  different 
computers,  the  computer  system  requirements  are  presented  separately  for  each 
of  the  three  programs — the  UI,  RUM,  and  MODCOM. 


USER  INTERFACE 

1.  Digital  computer  with  at  least  330  K bytes  of  problem  space  (exclusive  of 
operating  system). 

2.  ANSI1  FORTRAN  (or  higher)  compiler  for  computer. 

3.  Remote  input/output  (hard-copy)  terminal  with  at  least  72  characters  per 
line  and  a data  rate  of  at  least  10  characters  per  second.  (A  portable 
terminal  and  higher  character  rates  are  very  desirable.) 

4.  FORTRAN-eallable  subroutines  for  remote  terminal  input  or  output. 

5.  A medium  (card  punch,  magnetic  tape,  or  sequential  disk  file)  for  transmis- 
sion of  course  design  to  RUM. 

6.  Overlay  capability  or  conditional  job  steps  in  the  system  loader/ operating 
system. 

7.  The  computer  system  should  be  time-shared  with  the  capability  during 
normal  working  hours  to  begin  processing  90  percent  of  the  user  responses 
within  five  seconds  after  the  response  is  entered. 

8.  Disk  program  storage  of  at  least  800  K bytes. 

9.  Working  disk  sequential  file  storage  of  at  least  70  K bytes. 

RESOURCE  UTILIZATION  MODEL 

1.  Digital  computer  with  between  600  and  1200  K bytes  of  problem  space 
(exclusive  of  operating  system).  Core  requirements  can  be  reduced  by 
using  overlays. 

2.  SIMSCRIPT  II  (or  higher)  compiler  for  computer. 

3.  High  speed  printer  with  132  characters  per  line  and  a data  rate  of  at  least 
200  lines  per  minute. 

4.  A medium  (card  reader,  magnetic  tape,  or  sequential  disk  file)  for  recep- 
tion of  course  design  from  UI. 

5.  Disk  or  tape  program  storage  of  at  least  700  K bytes. 

1 American  National  Standards  Institute. 
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MODCOM 

1.  Digital  computer  with  at  least  162  K bytes  of  problem  space  (exclusive  of 
operating  system). 

2.  ANSI  FORTRAN  (or  higher)  compiler  for  computer. 

3.  Card  reader  (80  column  card)  with  a data  rate  of  at  least  100  cards  per 
minute. 

4.  High-speed  printer  with  132  character  lines  and  a data  rate  of  at  least  200 
lines  per  minute. 

5.  Disk  or  tape  program  storage  of  at  least  190  K bytes. 


